Papers with multimodal sentiment analysis
Self-Supervised Unimodal Label Generation Strategy Using Recalibrated Modality Representations for Multimodal Sentiment Analysis (2023.findings-eacl)
Copied to clipboard
| Challenge: | Multimodal sentiment analysis (MSA) has gained much attention over the last few years due to a lack of unimodal annotations in benchmark datasets. |
| Approach: | They propose a framework which integrates multimodal and unimodal tasks to optimize learning representations from multimodal data. |
| Outcome: | The proposed model learns to weight features differently based on features of other modalities and auto-generates unimodal annotations via a unimodule. |
Missing Modality meets Meta Sampling (M3S): An Efficient Universal Approach for Multimodal Sentiment Analysis with Missing Modality (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing methods to address missing modalities often assume a particular modality is completely missing due to recording or transmission error. |
| Approach: | They propose a missing modality-based meta-sampling approach for multimodal sentiment analysis with missing modalities . they conduct experiments on IEMOCAP, SIMS and CMU-MOSI datasets . |
| Outcome: | The proposed method significantly improves on existing models with a mixture of missing modalities. |
Multimodal Language Analysis with Recurrent Multistage Fusion (D18-1)
Copied to clipboard
| Challenge: | Comprehending multimodal language requires modeling interactions between modalities and between them. |
| Approach: | They propose a multistage fusion network which decomposes the fusion problem into multiple stages, each focused on a subset of multimodal signals for specialized, effective fusion. |
| Outcome: | The proposed model performs state-of-the-art across three datasets relating to multimodal sentiment analysis, emotion recognition, and speaker traits recognition. |
MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences (2021.naacl-main)
Copied to clipboard
Jianing Yang, Yongxin Wang, Ruitao Yi, Yuying Zhu, Azaan Rehman, Amir Zadeh, Soujanya Poria, Louis-Philippe Morency
| Challenge: | a novel graph-based neural model for multimodal sequential data is proposed . fusion is the process of blending information from multiple modalities, usually preceded by alignment . |
| Approach: | They propose a graph-based neural model that converts unaligned data into a modal-temporal graph . they use a dynamic pruning and read-out technique to efficiently process the graph fusion operation . |
| Outcome: | The proposed model performs state-of-the-art on multimodal sentiment analysis and emotion recognition benchmarks while utilizing significantly fewer model parameters. |
Mixture of Multimodal Adapters for Sentiment Analysis (2025.naacl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have been used for text sentiment analysis but sentiment is hidden in other modalities. |
| Approach: | They propose to fuse emotions from different data to analyze sentiments . they use compression parameter for each expert to reduce training burden . |
| Outcome: | The proposed method achieves state-of-the-art with a tiny trainable parameter count compared to current methods . emotions hidden in body movements or vocal timbres eclipse traditional methods compared with text sentiment analysis . |
Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis often fail due to equipment failure, data corruption, privacy issues and the like. |
| Approach: | They propose a multimodal Transformer framework using prompt learning to address the issue of missing modalities. |
| Outcome: | The proposed framework outperforms existing methods significantly across evaluation metrics. |
Denoising Bottleneck with Mutual Information Maximization for Video Multimodal Fusion (2023.acl-long)
Copied to clipboard
| Challenge: | Prior denoising methods suppress redundant and noisy information at risk of losing critical information. |
| Approach: | They propose a denoising bottleneck fusion model for fine-grained video multimodal fusion . they employ a bottleneck mechanism to filter out noise and redundancy with a restrained receptive field . |
| Outcome: | The proposed model improves on state-of-the-art video multimodal fusion benchmarks. |
CLGSI: A Multimodal Sentiment Analysis Framework based on Contrastive Learning Guided by Sentiment Intensity (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have focused on contrastive learning, but lack detailed learning of the distribution of sample pairs with different sentiment intensity differences in the contrastive training representation space. |
| Approach: | They propose a framework for multimodal sentiment analysis based on contrastive learning guided by sentiment intensity (CLGSI) it selects positive and negative sample pairs based upon sentiment intensity differences and assigns corresponding weights accordingly. |
| Outcome: | The proposed framework extracts common features between different modalities and then uses them to predict sentiment intensity. |
MemoSen: A Multimodal Dataset for Sentiment Analysis of Memes (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent studies on sentiment analysis of memes have focused on English, but there is a significant barrier to performing multimodal sentiment analysis research in resource-constrained languages like Bengali. |
| Approach: | They propose to use a Bengali dataset to perform multimodal sentiment analysis in low resource languages. |
| Outcome: | The proposed dataset for Bengali contains 4417 memes with three annotated labels positive, negative, and neutral. |
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors (P18-1)
Copied to clipboard
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, Louis-Philippe Morency
| Challenge: | Multimodal research is a growing field of artificial intelligence, and fusion is one of the main research problems. |
| Approach: | They propose a low-rank multimodal fusion method which integrates multiple unimodal representations into one compact multimodal representation. |
| Outcome: | The proposed method achieves competitive results on multimodal sentiment analysis, speaker trait analysis, and emotion recognition tasks while reducing computational complexity. |
Curriculum Learning Meets Weakly Supervised Multimodal Correlation Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have used the correlation information stored in samples for self-supervised learning, but they feed the training pairs in a random order without consideration of difficulty. |
| Approach: | They propose to inject curriculum learning into weakly supervised multimodal correlation learning by scoring and feeding pairs according to difficulty. |
| Outcome: | The proposed model achieves state-of-the-art on multimodal sentiment analysis without human annotation. |
Integrating Multimodal Information in Large Pretrained Transformers (2020.acl-main)
Copied to clipboard
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, Ehsan Hoque
| Challenge: | Recent Transformer-based contextual word representations have shown state-of-the-art performance in multiple disciplines within NLP. |
| Approach: | They propose an attachment to BERT and XLNet that allows them to accept multimodal nonverbal data during fine-tuning. |
| Outcome: | The proposed attachment allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. |
Cross-lingual Multimodal Sentiment Analysis for Low-Resource Languages via Language Family Disentanglement and Rethinking Transfer (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing multimodal sentiment analysis methods are limited to textual data and cannot handle multimodal scenarios. |
| Approach: | They propose a transfer learning framework that allows cross-lingual and cross-modal alignments and a language family disentanglement module that enhances the sharing of language universals within families. |
| Outcome: | The proposed method is superior to existing methods and can handle low-resource languages. |
CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies in multimodal sentiment analysis only use unified multimodal annotations, which do not reflect the independent sentiment of single modalities. |
| Approach: | They propose a Chinese single- and multi-modal sentiment analysis dataset with multimodal and independent unimodal annotations that can be used to study the interaction between modalities. |
| Outcome: | The proposed methods achieve state-of-the-art performance and learn more distinctive unimodal representations. |
A Text-Centered Shared-Private Framework via Cross-Modal Prediction for Multimodal Sentiment Analysis (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies treat all three modal features equally and implicitly explore the interactions between different modalities. |
| Approach: | They propose a text-centered shared-private framework for multimodal fusion . they propose modalities that can provide shared and private semantics . |
| Outcome: | The proposed framework outperforms baselines on the MOSEI and MOSI datasets. |
ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis (2023.acl-long)
Copied to clipboard
| Challenge: | Multimodal sentiment analysis aims to predict the sentiment of video content. |
| Approach: | They propose a framework that performs contrastive representation learning and contrastive feature decomposition to enhance the representation of multimodal information. |
| Outcome: | The proposed framework outperforms baseline methods on CH-SIMS, MOSI and MOSEI datasets on a range of metrics. |
Listen, Watch, and Learn to Feel: Retrieval-Augmented Emotion Reasoning for Compound Emotion Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to assess human emotion are limited by the subjective nature of emotion perception, limiting the robustness of existing models. |
| Approach: | They propose a plug-and-play module that enhances MLLMs’ ability to tackle compound and context-rich emotion tasks. |
| Outcome: | The proposed framework improves MLLMs' ability to tackle compound and context-rich emotion tasks and the Compound Emotion QA dataset shows it performs well across both benchmarks and evaluation frameworks. |
AMOA: Global Acoustic Feature Enhanced Modal-Order-Aware Network for Multimodal Sentiment Analysis (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods treat three modal features equally, without distinguishing the importance of different modalities. Existing models split the video into frames, leading to missing the global acoustic information. |
| Approach: | They propose a global Acoustic feature enhanced Modal-Order-Aware network to address these problems. |
| Outcome: | The proposed model outperforms state-of-the-art models on two public datasets. |
Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on multimodal sentiment analysis relies on back-propagated task loss or geometric property of feature spaces to produce favorable fusion results. |
| Approach: | They propose a framework which hierarchically maximizes the Mutual Information (MI) in unimodal input pairs and between multimodal fusion result and unimod input to maintain task-related information through multimodal integration. |
| Outcome: | The proposed framework maximizes the Mutual Information (MI) in unimodal input pairs and between multimodal fusion result and unimodulated input to maintain task-related information through multimodal integration. |
ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysis (2023.findings-acl)
Copied to clipboard
Yakun Yu, Mingjun Zhao, Shi-ang Qi, Feiran Sun, Baoxun Wang, Weidong Guo, Xiaoli Wang, Lei Yang, Di Niu
| Challenge: | Existing methods for multimodal sentiment analysis focus on general knowledge, which is inadequate to identify specific sentiments across modalities. |
| Approach: | They propose a method where specific-knowledge representations for each modality can be learned together with general knowledge representations via knowledge injection based on an adapter architecture. |
| Outcome: | The proposed method outperforms all prior methods on three popular benchmarks on multimodal sentiment analysis metrics. |
Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focus on equally treating the contribution of each modality or statically using text as the dominant modality to conduct interaction, which neglects the situation where each modal may become dominant. |
| Approach: | They propose a Knowledge-Guided Dynamic Modality Attention Fusion Framework (KuDA) that uses sentiment knowledge to guide the model dynamically selecting the dominant modality and adjusting the contributions of each modality. |
| Outcome: | The proposed model can be used to highlight the contribution of dominant modality through the correlation evaluation loss. |
Improving Multimodal Sentiment Analysis: Supervised Angular margin-based Contrastive Learning for Enhanced Fusion Representation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis focus on forming positive and negative pairs, neglecting the variation in sentiment scores within the same class. |
| Approach: | They propose a framework to enhance discrimination and generalizability of the multimodal representation and overcome biases in the fusion vector’s modality. |
| Outcome: | The proposed model improves discrimination and generalizability of the multimodal representation and overcomes biases in the fusion vector’s modality. |